Trevor Page for Software Testing
5 Labs (10% Each) – 50% 2 tests (25% Each) – 50% One written knowledge based test One hands-on Software testing evaluation Meet twice a week 2 hour lecture – knowledge and testing techniques 2 hour lab – Demos and time to work on labs Software Testing Is a growing field and is becoming extremely important as software touches more of what we do on a daily basis As the complexity of software increases, testing has become more and more important An appreciation for testing will result in a programmer producing better code Many entry level positions software developers will be hired as testers first Software Testing: Is an Empirical/Methodical investigation conducted to provide information on a Product Quality Assurance (QA): Systematic Monitoring and evaluation of different aspects of a product to ensure that standards of quality are being used What is the goal of a Software Tester? To find bugs! To find bugs as early as possible in the Software development process and to make sure that they get fixed (in a reasonable time) What is the goal of a person involved in Quality Assurance? Ensure that process and procedures are followed and to improve the process with the goal that process improvements will result in improving the software Will reduce the probability of bugs getting into the software in the first place. Are programmers responsible for any of the activities involved in Software Quality and Testing? What are they? Yes Build code that is well-documented that follows best practices and provide proof that the code meets the specification What is ISO? International Standards Organization What is ISO 9000? Standard for Quality Assurance Name the two sections of the ISO standard that apply to Software Development ISO 9001 ISO 9000-3 What are the two main goals of the ISO 9000 process? Targets the development process – how does an organization complete its work Tells an organization that the process requirements are – What to do (not how, or verify that it is done well) Provide some of the attributes of High Quality Software: Accurate/Complete Documentation UI follows standards for the device/OS Behaves: Accurately, Predictably, Reliably, Consistently Does what the user/specification says it should do What are soft the different job titles associated with Software Testing? Test Technician Test Engineer Software Test Toolsmith Test Lead – Test plan developer/manager Test Manager – Overseas all testing within an organization What are some of the important traits that a good software tester should have? Like to Explore Likes to troubleshoot Relentless Creative – Develop tests that will expose problems “Somewhat” of a perfectionist Tactful and diplomatic Persuasive – Needs to convince people of the importance of correcting the bugs List some of the main tools and documents that a software tester will use to accomplish his job successfully Plan – an overall document that describes what will be done during testing A list of test cases (use cases are often used to build test cases) A method/system to document the bugs/problems Testing tools and automation Metrics/Statistics/Summaries – Managers need these List four software development life cycle models and where testing fits in: Big-Bang Model – Code based on a quick description Testing: Little – Usually done by the end-user Code & Fix – Useful for small utilities Testing: Code – Test – Fix – Iterate until acceptance Waterfall – Thorough specification Testing: A complete phase of testing occurrence once development is complete Agile-Spiral Method Testing: Test phases are integrated throughout the development cycle. What are some of the realities or limitations of testing software that all parties involved in the development or use of software must be aware of: Not all bugs will be found / fixed in any significantly large system. You can’t write tests for all cases Some bugs are too risky to fix It may not be worth it to fix the bug – Limited number of users, simple workaround Reference Material Patton Ch 6 (pgs 91-104) Java Style programming Guidelines Provide a list of the ways that a standard or programming guideline applies to software development: Naming General Readability Statement Order – Class Design General programming rules E.g. – positive vs negative logic Comments – in-line vs Doc style Provide a list of considerations when developing a software solution? Technology to use – supportable Performance/Scalability Specifications of Client/End-User Programming Guidelines/Quality process When different considerations conflict, how are they resolved? Specification should be the guiding document followed by guidelines Solution still needs to be developed! Under some circumstances, process may not be followed in order to complete the development When you review code what type of testing is this referred to as: White-box testing or Clear box testing Referred to as white box or clear box because we cans see the code we are testing Provide a list of techniques that can be used to review program code: Peer review General informal review process Walkthrough More systematic/detailed review of work Inspection Fairy formal – review of application of standard / specific guidelines Items to review / check during a code review: Data Reference Errors Data Declaration Errors Computation Errors Comparison Control of Flow Parameter Errors Input / Output Are these solved when using Java/.NET (modern Dev Environments) ** - Often requires more than a code review – Formalized testing Data Reference Errors Mostly automatic – watch warnings Data declaration Errors Look for warnings Computation Errors** Needs a manual review / proof Comparison Semi-Automatic Control of Flow Sometimes automatic (loop entry/exit/no path to statement) Parameter Errors** Matching types – mostly automatic, units of measure can still be a problem Input / Output** Needs testing Provide a list of four different comment types that should be included in newly developed software: Overall Descriptive Code Explanation Documentation Domain / Business Describe what should be contained in Overall / Descriptive comments: What does the module / component accomplish from a high level Typically at the top of a module Typically at the top of a module Describe what should be contained in Code Explanation Comments: Details on ‘non-trivial’ code that mays be unclear to a junior level programmer within that environment. Well-written code should be somewhat self-explanatory. Describe what should be contained in Documentation Comments: Extractable comments by programmer tools (JavaDoc/Ndoc/Doxygen) Description of how class/method should be called and expectation of parameter values Describe what should be contained in Domain / Business Comments: Explains the business rules that have been implemented in the code, what and why and references Reference Material Chapter 7 – Pg 105-122 Junit Tutorial http://netbeans.org/kb/docs/java/junit-intro.html#Exercise_30 List ways a programmer can test the code he/she has developed: Code Trace A Code Trace is a static analysis that is done by viewing the code Debugging to understand/manipulate operation Develop Code to execute specific tests Define Data Coverage Testing that employs a number of different data combinations that will ensure that the testing is complete. 100% data coverage indicates that ALL combinations of data sets have been supplied to the item being tested** Define Code Coverage Testing that will ensure that all code paths have been executed at least once during the testing. NOTE: Adequate Data Coverage should yield 100% code coverage Provide a definition of a unit test A unit test is a test specific case or data set that tests a function (method) or confirms the state of a component (class instance – object) A unit test is constructed as a method of a special testing class that is not part of the application, but is included as part of the project. Code -> Compile -> Debug -> Test -> Run---| ---------------Correct----------------------| Debug – Focused on one area… Fix something and re-compile. Add “test” step after Debug, its automated by unit testing, and will make sure your fix, hasn’t broken something else. What is the goal of unit testing? Confirmation of valid operation – Use cases Test to pass There may be many unit tests to confirm variations in operation of a single method Confirmation of expected error conditions Test to fail Often requires many test cases In the Java programming language what tool is used to provide unit testing? Junit Currently at version 4.x What is an assertion? An assertion is a way to confirm that a return value or object variable contains a value that is known to be correct. Example: Object reference should never be null How are assertions used when developing unit tests? Assertions are placed into unit tests to confirm an expected result. If the assertion results in a true value, the test is said to pass A false return value from the Assertion will result in a failure What is the name of the standard java library that implements assertions? Java.org.junit.Assert Import static org.junit.Assert.*; Provide an example of an Assertion that could be used for testing purposes? Public void testSetHour(){ Int hour = 12; Time instance = new Time(); Instance.setHour(hour); Int ohour = instance.getHour(); //Confirm that values are the same assertEquals(ohour, hour); } What is the purpose of the fail function? The fail function in unit testing is used to exit the testing in a failed state based on getting to a specific line of code that should never have occurred. Often used in expected exception cases. List the tags (annotations) that are used within java test source files that are used to indicate test methods. @Test @Ignore A valid test that should not be used at the current time. Often the test results in a condition that causes severe failure Database deletion/corruption @BeforeClass, @AfterClass These annotations are used to denote methods that will be called before and after the test class is invoked @Before, @After These annotations are used to denote methods that will be called before and after each unit test is invoked. Useful for common factoring common code for the unit tests Definition of a Unit Test Unit testing is a method by which individual units of source code are tested to determine of they are fit for use A unit is the smallest testable part of an application Goal of Test Case Selection: Use test data sets that will produce examples of ALL expected results with a minimum (practical minimum) of number of data sets Non-Normal Data Set Properties Bad data will result in a documented error result Bad Data will result in a default manner that is normal Null arrays produces a NullPointerException Bad Data will result in reproducible errors that can be trapped Example of non-Normal datsets Empty {} no values provided Null An object is not initialized Out of range Value is outside the normal range of the input data set Example: double result = Math.log(-10.0); Out of range for current object instance 3D game example where an object is created but not as yet placed into the world and a request is made to return the current position More difficult to validate Define testing automation Any mechanism that can be executed automatically with little/no user interaction. Report will be produced that summarizes the results of the test Junit with Netbeans is an automated test mechanism. What is regression testing? Running a series of previously successfully passed tests against a new version/revision of the application or component List ways that test automation can aid the Tester and the testing process. Speed – Tests can be executed without user intervention Efficiency – Test can be run independently of other development activities – multi-task Accuracy/Precision – The test will be the same each time Relentless – Test tools do not get tired or bored Provide a list of test tools and examples Viewer/Monitor Program that will collect data that can be analyzed independently Wireshark – TCP/IP protocol analysis Integrated Debugger Driver A main test method that will execute the methods of a class to ensure correct operation Junit integrated within a development environment operates as a test driver Stress Tester Large data sets to push boundaries of application Load Tester Simulator to duplicate multiple / concurrent use of an application Typically needed for large web-sites / web-based applications Profilers can be used to identify bottlenecks within an application Interface Injectors Modifies Data sets / Communication conditions to simulate real-world noisy conditions Name two types of regression tests and there application in an automated environment. White-Box unit testing All updates/revisions are verified against a set of unit tests Can be integrated into the development process that will prevent revisions be accepted by the source control system unless all unit tests pass Black-Box Integrated Test Result capture of integrated application is compared against a previous capture – tools like diff (unix) Considered to be a compare against a GOLD-STANDARD Name a method of testing automation for testing a User Interface? Macro-Record / Playback Macro Magic is an example List some realities to keep in mind when using Test Automation Software may be changed / improved Some tests may be invalidated Time spent on Testing (writing tests) will take time away from development Invasive Testing may impact a real-running system. If an automated tool finds a bug, try to reproduce by hand to confirm Test Plans and Test Cases Reference Material -Patton Ch 17, 18 (pgs 263-290) Why is Planning for your test effort important? The goal of planning provides a way to identify Requirements Resources Scheduling Makes team (developers/management) part of the process by informing them what is to be done. Provide a list of the important criteria when coming up with a test plan. What is the purpose of the plan What is to be tested, to what degree or level What are the quality goals of the Product Must have/nice to have Max # of bugs found in the last period Define Entry and Exit Criteria for the testing plan process ENTRY Under what conditions can testing start Feature completeness EXIT When does testing stop? Quality achievement level has been obtained Scheduling requirement – Deadline cannot be missed Since the goal of the test plan is to find bugs, define the four bug classifications that you may come across during testing: Software does not do something specified Software does something that the specification says it should not do Software does something that the specification does not mention Software does not do something it should**, that is not mentioned in the specification **Missing a feature that is present in similar software What should happen to bugs that are found that do not fit into the standard categories Any unspecified/incorrect behavior should be reported Simple – Inconsistent formatting / spelling Major – Blue screen of death, Data loss List the items that should be required for a Test project People Equipment Office Space Software/Tools specific for the testing Reporting/Tracking software Outsourcing companies Miscellaneous Supplies Reference material, training material (domain knowledge) For small projects – which items from the previous list are most important People Software/Tools Equipment Since Testing must be integrated into the SDLC process, list the two ways the scheduling could be placed into the project Hard Date scheduling Fix start and stop dates Advantages Eases scheduling / resource allocation Only real solution for outsourcing situations Disadvantages Difficult to control the entry and exit conditions from a quality perspective Delays can compromise the testing process Relative Dates Advantages Start date not set until “entry Condition” for the test cycle is met. Original estimates for testing time should be closer to reality Disadvantages Shifted time may not optimize resources Testing team may be allocated to other projects in future and may still end up in a time crunch Test Plan – Overall description of exactly what will be covered as part of this cycle Test Design – Use cases should be used as the basis for tests Test Case(s) – An example of a use case (part of a use case) that can be run against the software – setup Test Procedure – A step-by-step description of “exactly” what should be done, and what is expected as a result Define test case planning Test case planning takes a test plan and reduces it down to a set of scenarios (use cases) that describe exactly what and how the software or components is to be tested Clearly defines the condition and the expected result Not limited to unit testing List four reasons that test case planning is important Organization – Provides a way to scope/schedule the testing Repeatability – the test case should produce a case that is reproducible Tracking – The test case will describe a report mechanism for feeding back the results of the report Proof – Captured results can be used as proof that the software works or does not work under a specific set of conditions Important for particular industries that must prove compliance for safety reasons What is a test case identification? A way to uniquely identify a test case Why is test case identification important? Provides a mechanism to discuss/communicate the status of a particular scenario Test case naming can be automated, but is often useful when reporting issues into a bug tracking system Provide a list of attributes that may also require specification in a test case: Input Conditions Special procedures to get software to the start condition Data set or database condition Expected output conditions Environmental set-up What other integrated components must be installed / started. What state must they be in What is the purpose of the Test Procedure? Allow other testers to conduct the test All good test procedures should be clearly stated so that the test is reproducible and the result obtained is also reproducible Bug Reporting and Resolution & Web Site Testing Reference Material – Ch 19 (pgs 291-311), Ch 14 (pgs 211-227) Provide a list of reasons that are often used to delay or skip fixing a bug. Not enough time Not really a bug Too Risky to Fix Not worth it to fix – impacts few users Bug report is vague – difficult to reproduce Provide two important aspects of bug reporting that will increase the chances of the bug being corrected: Report as soon as possible during the testing cycle Effectively describe the bug Provide a list of attributes that will create an Effective Bug description Provide information that is relevant to the bug. OS Version, Software Build One bug / report Extra information can cause confusion Document the process to duplicate Procedure Before reporting a bug, provide a list of steps that will validate the state of the bug. Reproduce the bug Write up the steps required Attempt the steps as documented to verify that it is a bug Why is following up on a bug report important? To ensure that it gets fixed or moved from the open state with a proper explanation Provide some criteria that should be used to decide the order of bug fixes: Severity System crash, Data Loss, security breach Loss of functionality, wrong result Formatting, rare event Priority Impacts the testing process Bugs that interfere with the testing process are considered to be high priority bugs. Example: User attempts to log-in as an administrator, but is unable to do so. This will block testing on all admin functions Must fix Should fix if time permits Would like to fix but the product can still be released Describe the states of a bug during a typical bug life cycle. OPEN – First reported into the system REVIEW / CONFIRM – Analyzed to verify RESOLVED – Fix applied DEFERRED – Recognized, but bug is a low priority – may not be corrected in this version CLOSED – Bug has been corrected and solution has been accepted Provide some reasons why web-sites need to be tested: Web site can have a large audience They are public Will reflect on the company Need to be professional – Can have a diverse set of users / uses How can you apply black-box testing to a web-site? Black-box testing applies directly to web-sites from a usability point of view NO docs – simply test Does the web-site follow standard conventions? Can it be navigated successfully? Content can be tested by reading the content – Time consuming, but difficult to manage in other ways Provide a list of items that need to be tested on a web-site and a manual way to perform the test: Graphics/Text – Read the text / confirm graphics matches context (graphics appear) Hyperlinks – Follow the links Layout Check – Reasonable browser re-sizing to confirm resizing of the site Forms – Form data entry works as expected Appropriate error messages generated What is Gray box-testing? Partial view into the code on a web-page HTML is visible CSS/Javascript can also be visible What types of items should be reviewed in a gray-box test? Use of standard / modern tagging Use of outdated features – These are not always well supported in new devices Information that should not be in production code Comments What is Configuration and Compatibility testing? Testing of site by using a combination of different hardware and software This type of testing is important in a fractured hardware/software environment http://www.netmarketshare.com Provide a list of items you should try on a standard web page to check for compatibility Hardware platform – mobile market Browser Browser version can also be an issue Plug-ins If site relies on plug-ins, are they widely available Speed Not as much of an issue today (~10% till use dial-up) How can automation help with Web-Site testing? Reduce time on testing. Focus testing List tools that can be used to automate some aspects of web-site testing: Link checking Spell Checking Performance checking Look at response times COMP-10066 Mid-term review March 9, 2011 – 8am – A224 90 Minutes in length 25% of course mark All Written Examples are Java based 1) Software Quality – Lab #1 Code Rules (language specific or general) Process Software testing, Where does it fit? Black-Box White-Box Static/Dynamic testing Test-to-pass Test-to-fail Data coverage Code coverage Unit testing Jobs in software testing What is the rule of the tester Bugs/defects Test Data – Unit testing What defines a good data set for a particular method Test plan – theory – Function of plan Bug resolution & reporting – theory 60 marks – 5 questions – Definitions (10) Diagrams Testing in sdlc Bug life cycle Cost vs time Practical Present code – ask to suggest improvements Specific to example Bugs and priority Unit testing data sets